Large Scale Data Mining: Challenges and Responses

Data mining over large data-sets is important due to its obvious commercial potential, However, it is also a major challenge due to its computational complexity. Exploiting the inherent parallelism of data mining algorithms provides a direct solution by utilising the large data retrieval and processing power of parallel architectures. In this paper, we present some results of our intensive research on paralielising data mining algorithms. In particular, we also present a methodology for determining the proper parallelisation strategy based on the idea of algorithmic skeletons and performance modelling. This research aims to provide a systematic way to develop parallel data mining algorithms and applications. ’ Parallelism in Data Mining Algorithms Numerous algorithms have been previously developed for data mining (Fayyad, Piatetsky-Shapiro, & Smyth , nnc\ n-1...:-1. A ̂ ,....^^ -P :.,L.,,,,* -nnn ,,,,;nm ^“;,+;,,,. IYYU). I Ilt: llCll uc:grcr; “I IIIIItxGIIb palculGlmul LuDUllltj in these algorithms allows some flexibility in choosing the parallelisation scheme that most suited for a particular parallel machine. Two major methods for exploiting parallelism within data mining algorithms can be identified as task parallelism and dafa parallelism. A brief summary of how the two approaches can be used to parallelise different data mining algorithms is given in Table 1. In the task parallelism approach the computation is partitioned amongst the processors of a parallel machine with each processor computing a distinct part of a learning model before co-ordinating with the other processors to form the global model. In the data parallelism approach the training set is partitioned amongst the processors with all processors synchronously constructing the same model; each operating on a different portion of the data-set. Our practical experience with parallelising data mining algorithms showed an interesting phenomenon. While the parallelisation of certain data mining algorithms shows a consistent performance behaviour when ’ &pyJvrieht fi31997. .A_r?leric~A_ssc&&ion for Artificial o-sv> Intelligence (www.aaai.org). All rights reserved. Algorithm